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Abstract 


This paper describes a study, which had as its principal goal the development of a sufficiently 
detailed 5x5 Risk Matrix Scorecard. The purpose of this scorecard is to outline the criteria by 
which technical issues can be qualitatively and initially prioritized. The tool using this score 
card has been proposed to be one of the information resources the NASA Engineering and Safety 
Center (NESC) takes into consideration when making decisions with respect to 
incoming information on safety concerns across the entire NASA agency. The contents of this 
paper discuss in detail each element of the risk matrix scorecard, definitions for those elements 
and the rationale behind the development of those definitions. This scorecard development was 
performed in parallel with the tailoring of the existing Futron Corporation Integrated Risk 
Management Application (IRMA) software tool. IRMA was tailored to fit NESC needs for 
evaluating incoming safety concerns and was renamed NESC Assessment Risk Management 
Application (NARMA) which is still in developmental phase. 
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1.0 Background 


The NASA Engineering and Safety Center is an independent organization, chartered in the 
wake of the Space Shuttle Columbia accident and in response to the observations of the 
Columbia Accident Investigation Board (CAIB). The objective of the NESC is to ensure 
the safety and engineering excellence of NASA's programs and institutions. This is to be 
accomplished by using a diverse mix of talent from across the nation for the following 
activities: coordinating and conducting robust engineering testing and safety assessments; 
performing independent testing and analysis; and reviewing programs and institutional 
issues at all centers including headquarters. 

The organization will be proactive in looking for the next problem or issue as well as 
be one of several collection points for all identified safety related reports, issues, 
problems and/or concerns for the entire NASA agency. The product of the NESC 
will be a formal, independent technical assessment of all major NASA programs 
resulting in written reports, briefings, and/or recommendations. Additionally, the 
NESC will provide a centralized location for the management of independent 
engineering assessment by expert personnel and state of the art tools and methods 
for the purpose of assuring safety. 

Areas for independent assessment will be carefully selected and managed with a 
strong focus on technical merit. Work will be prioritized based on technical risk, the 
need for independence and the potential for value-added contribution. Methods for 
this selection and prioritization are ongoing and include the tasks completed during 
the contract efforts described in this report. 



2.0 Risk Matrix Background 


Widely accepted and state-of-the-art risk matrix templates are available from several 
reputable sources including, Defense Acquisition University (DAU)/Defense Systems 
Management College (DSMC) 1 , Software Engineering Institute (SEI) 2 , NASA Policy 
Guidance Document 7120.5B 3 as well as others. The critical step in establishing a 
meaningful and useful risk management and risk analysis system is tailoring the risk matrix 
components. This is accomplished by weighing the program’s mission, goals and 
objectives while concurrently considering the organization’s degree of risk affinity or 
aversion. The objective of evaluating the attributes of a risk is to gain a better 
understanding by determining the expected impact, probability and timeframe of the risk. 
Risks should be evaluated at a level of analysis that is sufficient to determine the relative 
importance for effective handling strategies. 2 For NESC purposes, this step will be used 
in the decision if action will be taken and with respect to who has the responsibility of 
assessing the issue. 

The effort of tailoring a 5 x 5 Risk Matrix Scorecard to NESC needs included 
developing both quantitative and qualitative attributes and definitions. This will 
provide a framework for communicating how individual issues will be initially 
classified and later prioritized by NESC. The 5x5 Risk Matrix Scorecard 
developed for NESC use is shown in Figure 1. The following sections explain in 
detail how each element of that scorecard was developed. 

3.0 Risk Definitions 
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Following two days of discussions on risk definition, it was agreed that, Fell 4 encompassed 
the mood toward risk that the NESC currently has adopted: 

“Unfortunately, there are no generally accepted definitions of the terms used in risk 

assessment shortly after its formation in 1981, the United States Society for Risk 

Analysis established a committee to define risk. After 3 or 4 years of work the committee 
published a list of 14 candidate definitions and reported that it could not reach agreement. 
They recommended that a single definition of risk not be established but that every one be 
free to define it as appropriate to his or her own work.” 

The development of the risk definitions for NESC purposes certainly does not 
attempt to change this situation; however, it is felt that the various terms apply to the 
NESC use. The following definitions were proposed: 

3.1 Risk 

Measure of the potential inability to achieve overall program objectives within 
defined constraints and has two components: 1) the probability /likelihood of failing 
to achieve a particular outcome, and 2) the consequence/impact of failing to achieve 
that outcome 1 . 

3.2 Consequence 

Impact (typically categorized as negative) to program and/or project (loss, injury, 
disadvantage). 

It was felt this definition of consequence was broad enough, combined with the 
scales developed for the matrix, to communicate NESC preferences. Keeping in 
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mind, NESC will be receiving risks, problems, concerns, and issues from several 
sources and various pipelines. 

3.3 Likelihood 

Ordinal scale: Relative ranking of probability of occurrence. Numerical scale: 
estimated range of probability an event will occur combined with the uncertainty in 
the probability assessment. 

When developing an NESC likelihood definition, it was important to demonstrate an 
ordinal scale, to clearly establish preference as well as a numerical scale. There seems to 
exist two schools of thought regarding how to communicate and assess the likelihood of 
the issues. One prefers an ordinal scale that represents only priority order and cleanly 
communicates priority between likelihood categories to those reviewing how issues have 
already been assessed and placed in the 5 x 5 matrix. The other school of thoughts prefers 
a numerical scale feeling it is important to those involved in initially assessing the issue. 
The numerical scale proposed for this effort represents ranges on a lognormal scale. The 
lognormal scale was proposed in order to allow the communication of events that happen 
with an occurrence of “one in some “x” order of magnitude. Thus, it was decided the 
likelihood should incorporate both aspects of what would be communicated in the 5 x 5 
Risk Matrix Scorecard. For communicating preferences and guiding others in establishing 
their subjective likelihood assessments, it was felt examples associated with each 
likelihood range were necessary. These examples are discussed in Sections 4.4 and 4.5 

3.4 Risk Management 
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An organized, systematic decision-making process that efficiently identifies risks, 
assesses or analyzes risks, tracks and communicates risk and effectively reduces or 
eliminates risks to achieve program goals 5 . 

These definitions appear to embrace the NESC feelings toward risk at this time. It is 
not inconceivable for these definitions to change or be modified as the NESC 
matures. 

4.0 Likelihood Definitions 

A definitions box was developed to accommodate and explain the 5 x 5 Risk Matrix 
likelihood axis. Four categories of descriptors were chosen for explaining and defining the 
likelihood bins and associated ranges. 

4.1 Level 

An ordinal scale from one to five was chosen. This ordinal scale simply represents the 
degree of probability rank ordered from one to five. Ratio between numbers is not 
intended to communicate anything other than preference in rank order from one level to 
the next. The least likely indicator is 1 and the most likely indicator is 5. There should 
not be any type of mathematical operations performed or associated with this ordinal 
scale. 


4.2 Probability 

The narrative probability scale is intended to be a brief descriptor of subject attitudes 
toward each likelihood descriptor. The range chosen was from “Very Low” to “Very 
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Likely”. Many studies have been conducted on the verbal ranges; however, nearly all of 
those found were for base ten numerical scales. The verbal likelihood descriptors had to be 
adopted for not only a lognormal scale, but also for the specific lognormal ranges 
proposed for NESC in this effort. For each of the ordinal scale categories, there is a 
corresponding probability description, as is the case for the following two categories. 

4.3 Narrative Description and Numerical Ranges 

The narrative field approaches a better descriptor, and is another step in the maturity and 
tailoring of the likelihood definitions box. Numerical ranges on a lognormal scale are 
incorporated into this category. The two schools of thought are now both satisfied: there 
is both an ordinal scale and a numerical scale. 

4.4 Example 

A perspective introduced during the NESC Risk 5x5 Matrix development is that of 
describing likelihood using numerical probabilities that reflect the occurrence of an event 
on a per opportunity basis. The paradigm of ‘per opportunity’ appears to be unique (with 
respect to a risk matrix development) to the NESC Risk Matrix. An example of this can 
be given as fatal crashes per passenger airplane departure or fatal crashes per automobile 
trip. This communicates, once the opportunity actuates (the departure in this case) there 
is either failure or not (failure, being described as death, injury, failure to meet an 
objective, goal or mission). 

It is important to note, the categories do not contain equal ranges. The numerical ranges 
for each level were proposed for NESC prior to any research on failures per operational 
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opportunity were conducted. It was discovered the ranges clearly represented distinct 
classes of events, which will be described in more detail later in this report. 

In addition to the narrative example, it was perceived that listing an event example for 
each range to which a layperson could comprehend would help those assessing the issues 
develop a better frame of reference. This is intended to facilitate the reduction of the 
uncertainty in subjective assessments of likelihood for this 5x5 matrix. 

4.5 Communicating Definitions within Matrix 

The following categories are distinguished by Level and Probability descriptors that 
correspond to the input parameters in the likelihood box on the 5x5 Risk Matrix 
Scorecard. Examples that fall into each category are represented in Table 1. 

4.5.1 Level 1 Probability Very Low 

Narrative: Likelihood of occurrence is estimated to be less than 0.000001 (10 6 ) per 
operational opportunity. 

Example: Fatal crashes per passenger airplane opportunity 6 . 

Brief Explanation : Air travel is by far one of the safest activities in which our society 
participates daily. It is a very mature technology in which we do not tolerate frequent 
failure. When failures do occur, we invest extensive resources to prevent future failures. 

4.5.2 Level 2 Probability Low 

Narrative: Unlikely but possible to occur. Likelihood of occurrence is estimated to be 
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between 0.000001 and 0.001 (10‘ 6 and 10‘ 3 ) per operational opportunity. 

Example: Fatalities per passenger vehicle trip 7, 8 . 

Brief Explanation: This category represents nearly all risks that are accepted by society as 
being acceptable to a layperson. 

Assumptions : In order to calculate a ‘per opportunity’ value, it was assumed a passenger 
vehicle trip was approximately 70 miles on average; a passenger bus trip was 
approximately 70 miles on average. These average miles per trip are for demonstration 
purposes only. It was felt, these average values were reasonable; however there was not 
any evidence available to the authors at the time of the writing of this paper verifying the 
average passenger vehicle trip, or the average passenger bus trip. 

4.5.3 Level 3 Probability Moderate 

Narrative: Likely to occur some time in the life of the item event. Likelihood of 
occurrence is estimated to be between 0.001 and 0.01 (10‘ 3 and 10‘ 2 ) per operational 
opportunity. 

Example: Fatalities per motorcycle trip 9 . 

Brief Explanation : This category represents events that our society appears to be divided 
on with respect to whether or not it is an acceptable risk. Many individuals make a 
conscious choice to avoid this activity; whereas some still accept the risk. 

Assumptions : In order to calculate a ‘per opportunity’ value, it was assumed a 
motorcycle trip was approximately 50 miles. Again, the value of miles per motorcycle trip 
is for demonstrational purposes only, but believed to be reasonable. 
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It should be noted; motorcycle trips are hover at the dividing line between level 2 and level 
3, depending on the assumed average miles per trip. If the assumed trip is approximately 
20 miles, motorcycle trips would be classified as level 2. 

4.5.4 Level 4 Probability High 

Narrative: Expected to occur some time in the life of the item. Likelihood of occurrence 
is estimated to be between 0.01 and 0.10 (10' 2 and 10 1 ) per operational opportunity. 
Example: US ELV Launches (1998 - 2001) and open-heart surgeries 10, 12 . 

Brief Explanation : Events in this category are still considered risky by society as a whole. 
The example provided, Expendable Launch Vehicles, does not involve human death. Or, 
if they do indeed involve human death, it is an accepted risk a person makes a clear and 
conscious choice for personal and/or health related issues. 

4.5.5 Level 5 Probability Very Likely 

Narrative: Likely to occur often. Likelihood of occurrence is estimated to be greater than 

0.10 (10 1 ) per operational opportunity 

Example: Mortality (death) during, or as a result of, brain surgery 12, 13, 14, 15 . 

Brief Explanation : It was discovered that real life examples in this category and probability 
range typically fall into three subcategories: 

1. Stupid stunts: Society as a whole does not accept death or injury rates in this 
category and we have evolved in such a way as to avoid such activity, unless it is 
out of sheer stupidity. Darwin Award Winners. 

2. Lesser of the Two Evils. There are instances where we will accept high failure 
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rates when options in this category are the only options available. Very risky 
surgeries are in this category. 

3. Easily/Cheaply replaceable items or, rarely re-used items. Examples are common 
things such as paper clips. This category is recommended for further investigation, 
especially in light of emerging technologies and ‘low use’ parts and/or systems. 

5.0 Consequence Definitions 

There was not any research performed in this category; but it is included for completeness. 
Comments and recommendations on this category are as follows: It is recommended, for 
consistency between likelihood and consequence values, consequence definitions receive a 
more in-depth study during future development of the NARMA software. In addition, 
there is an effort to ensure the color-coding scheme of the 5x5 risk matrix properly 
reflects not only NASA perspective, but NESC perspective as well. 

The Consequence Definitions evolved from Table 2 titled “Mishap Severity Categories.” 
The indicators for each category came from legacy information primarily obtained out of a 
Payload Classification Document. As a result of this information, NESC was quite 
comfortable with the descriptors in this table. The only value added by Futron was the 
replacement of “No” in level 1 with “minimal to no”. It was discussed and agreed upon 
“no” referred to the absolute value of zero and “minimal to no” communicated a more 
reasonable range for level 1 occurrences. 

It is to be noted, the descriptors discussed above and reflected in Figure 1 have been 
greatly abbreviated to accommodate the limited space available on the 5 x 5 Risk Matrix 
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Scorecard. 


5.1 Risk Consequence Scoring Terms 

It was felt that additional information might help reduce the amount of subjective 
assessment as much as possible. At this stage in the development of NARMA, the set of 
screening questions being developed by NESC in a parallel effort, have not been ‘tied’ to 
the risk consequence indicators, but this is something that needs to be done in the future. 
NESC is only dealing with issues that are considered technical in nature. Cost and 
Schedule considerations will not be evaluated. The emphasis is on the risk consequence 
scoring terms selected as described above. 

5.1.1 Safety, Health, Environment 

In the 5 x 5 matrix scorecard, Safety, Health and Environment (SHE) was selected as a 
consequence category for this effort because it was reflective of death, injury, or damage 
to resources. 


5.1.2 Mission Success 

Mission Success (MS) can be described as avoiding loss of crew (LOC), loss of vehicle 
(LOV) and/or loss of science mission (LOM). It is imperative that successful missions 
occur on a regular and frequent basis. There is a range of activities that fall under the 
umbrella of mission success and include the following: Major Mission Objectives 
(MMOs), Supplemental Mission Objectives, and costly hardware losses in a monetary 
sense. 
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5.1.3 National Significance 


NESC is concerned about events that fall into a category that has been titled National 
Significance (NS). National Significance can also be described as national prestige or 
visibility. At this critical point in NASA’s history, another high visibility failure of any 
kind is likely to result in public outcry and political fallout. The American public put great 
pride in the capabilities of NASA to achieve technological advances and feats. Public 
support is critical to future funding of NASA programs. 

5.1.4 Risk Scoring 

Currently, the NARMA software scores the risks by multiplying representative values for 
likelihood by consequence. This is done solely to obtain a draft qualitative prioritized 
listing of risks and an improved method is under development. Care must be taken with 
using multiplied values as measures. It must be noted, as an example, a score of “6” can 
be either 2 x 3 or 3 x 2, depending on the values attributed to each likelihood and 
consequence level. The first mathematical operation places the risk in the ‘green’ zone of 
the matrix and the latter places the risk in the ‘yellow’ zone. These two zones of the 
matrix, as described later may be treated quite differently by NESC in how they opt to 
disposition their risks. 

An alternative method is having a numerical value reflective of the ordered pair, 
likelihood, consequence (L,C) which would communicate more information such as either 
2,3 or 3,2 from the previous example. 
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5.1.5 Matrix Placement 


When determining risk consequence between Safety Health and Environment (SHE), 
Mission Safety (MS) or National Significance (NS), NARMA at the time of this paper 
uses the highest score represented among the three areas and places that value in the 
matrix corresponding to the consequence axis as shown in Figure 2. 

Future endeavors could involve creating individual placeholders in the matrix for S, M and 
N, as shown in Figure 3. This method will communicate more of the risk information. 

For the likelihood values, only one value can be chosen. If a risk assessor feels there are 
different likelihood values for the consequences of SHE, MS and/or NS writing a separate 
risk should be considered. It should be noted that currently NARMA is not capable of 
correlating or linking risks within the database. This capability has been proposed by both 
the International Space Station (ISS) program office as well as the Orbital Space Plane 
(OSP) program office (both of which have implemented tailored versions of IRMA) and is 
currently being investigated by Futron for a future version of IRMA. 

5.2 Risk Matrix and Legend 

A 5 x 5 risk matrix was proposed for the NESC purposes. It was decided that a typical 3 
x 3 did not provide enough resolution, anything greater than a 5 x 5 was too distracting 
and anything nonsymmetrical was not logical. 

The 5x5 matrix has been color coded with red, yellow and green segments for each 
matrix ‘value’. Currently the matrix color-coding is NOT symmetrical, but is under 
intense scrutiny. Individual likelihood combined with individual consequence needs to be 
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evaluated in detail. This is suggested as a future effort. 

A legend was developed to explain the color-coding within the 5 x 5 matrix. This color- 
coding is currently under scrutiny as will be explained in the following sections. 

5.2.1 Low 

It is proposed, those items color-coded green within the risk matrix be considered as ‘low’ 
by the NESC. It is also proposed, an item in this category be accepted by NESC for 
Board review and it is probable that a referral to another NASA Independent Assessment 
organization will be made. 

5.1.2 Medium 

It is proposed, those items color-coded yellow be considered ‘medium’ by the NESC. It is 
recommended an item in this category be accepted for NESC Board review. NESC or 
other NASA Independent Assessment group or organization action is required. 

5.1.3 High 

It is proposed, those items color-coded red be considered ‘high’ priority items. An item in 
this category may immediately be accepted for NESC Board review. It is most probable 
NESC will perform the follow-on independent assessment. 

6.0 Summary and Conclusions 

A 5 x 5 Risk Matrix Scorecard has been tailored for NESC purposes and incorporated 
into the NARMA software application. The NARMA NESC 5x5 Risk Matrix scorecard 
is shown in Figure I and can be accessed through the “Help” menu contained within the 
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NESC Assessment Risk Management Application (NARMA) which is the NESC database 
used to assess and prioritize concerns brought to the attention of the NESC. 

The selection of tools and methods to achieve the goals and objective of the NESC are 
maturing and ongoing. The tasks completed 1) directly support the efforts to ensure the 
independent assessments within NASA will be carefully selected and managed and 2) 
assist NESC in securing their goals and objective of ensuring the safety and engineering 
excellence of NASA's programs and institutions. The recommendations made for further 
development of these tools and processes are critical extensions of the work performed 
thus far. 
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NESC Risk Summary Card 


NESC RISK ASSESSMENT 



RISK 

DEFINITIONS 


Bisk: Measure of the potential inability to achieve overall program objectives within defined constraints and has two 
components: (1) the probability /likelihood of failing to ochieve a particula outcome, and (2) the 
consequences/impact of tailing to achieve that outcome 

Consequence : Inpact (typically categorized as negative) to progamjproject(loss, injury, disadvantage) 

Likelihood : Ordinal scale: Relative ranking of probability of occurrence. Numerical scale: estimated probability an 
event will occur corrtoined with the uncertainty in the probability assessment 


RISK MANAGEMENT : An organized, systematic decision -making 
process that efficiently identifies risks, assesses a analyzes risks, and 
effectvely reduces or eliminates risks to achieving progam goals. 
(NESC Risk Management Plan) 

NESC ASSESSMENT RISK MANAGEMENT APPLICATION (NARMA) :The 
N ES C database used to assess and prioritize concerns brought to the 
attention ofthe NESC. URL http:yfrx 


What is the likelihood the situation or cinzumshcrnce will happen? 


NESC RISK MATRIX 


Leiei 

Probability 

. . .or -.. . 

Example 

E 

Very likely 

likely to occur often, likelihood of occurrence is 
estimated to be geater than o.to (too) per 
operational opportunity 

Mortality Rate for 
Brain Surgery 

D 

High 

Expected to occur some time in the life of the item, 
likelihood of occurrence is estimated to be between 001 
and 0.10 (10-2 and 10-1) per operational opportunity 

Failures per US ELV 
Launch (1988 - 
2001) 

C 

Moderate 

likely to occur some time in the life of the item. Ukelihood 
of occurrence is estimated to be between 0.001 and 001 
(10-3 and 10-21 pet operational opportunity 

Fatal Crashes 
per motorcycle 
_trip ' 

B 

Low 

Unlikely but possible to occur, uloelihood of occurrence 
is estimated lobe between 0.000001 and 0.001 (10 -A and 
10-3) per operational opportunity 

Fatal crashes 
per automobile 
trip 

A 

Very Low 

likelihood of occurrence is estimated to be less than 
.000001 (<1 0 -A) per operational opportunity 

Fatal crashes per 
passenger airpbnr 



□ 


I High - Accepted for 
NESC Board review. 
Probable NESC follow- 
on I A. 

Medium - Accepted for 
NESC Board review. 

NESC or other NASA IA 
action required. 

1 low - Accepted for NESC 
^ Board Review - Protable 
referral to other NASA IA 


RISK CONSEQUENCE SCORING TERMS 


(Safety, Health, Environment, Mission Success, National Significance) of this NESC Risk? 


Safety, Health, Environment is defined as impact to life, health 
1 working environment and natural environment 


Level 

A 

B 

c 

D 

E 

Safety, 

Health, 

Environment 

Minimal/no safety 
or health plan 
violations y Minimal 
to no environ 
impacts 

Could result in 
injury or Nines not 
resulting in lost 
work day / 
Minimal ervimmlt 
damage 

could result in injury 
or occupational 
illness resulting in one 
or more lost work day 
1 Mit. ervirnmtt 
damage w/o law viol 

Could result in 
permanent partial 
disability, injuries or 
occupational illness 1 
Revefsibleenvinmt 
damage - violates la w 

Could result in death 
or perm, total 
disability i Irreversible 
severe environ 
damage that violates 
aw or regulation 

Mission Success definition includes inpacts to Major Mission 
2 Objectives (MMOs) as well ce hardware bss 

National Significance is defined as the degree to which national 
3 prestige, viability and public relations ae impacted 

Safety, Health Envronment Mission Success and National 
4 Significance can existconcurrently and are not mutually exclusive 


Mission 

Hardware loss 
between $200K 
and $1 Million 1 
Failure to anyone 
MMO 

Hardware bss 
between $lMand$lo 
Million t Failure to 
meet >507. of 
supplemnll objectves 

Hardware bss 
between $10M and 
$100 Million V Failure 
to meet any one 
MMO 

Hardware bss 
between $100Mand 
$250 Million y failure to 
meet>5CKMMO^ 

Hardware bss 
exceeding $250 
Million y Failure to 
meet all Major 
Mission Objectves 
fMMCjsl 

Risk scoring is accomplished by numerical value which is reflective 
C of the ordered pair L C . When determining risk oonsequence 
3 among S HE, MS and NS, the highest score is represented in Ihe 
N ES c Risk Matrix as a single score value . 

National 

Significance 

Minimal ar no 
identified 
National Prestige 
or Visibility 

Low National 
Prestige and Visibility 

Moderate National 
Prestige and Visibility 

significant National 
Prestige and 
Visibility 

High National 
Prestige and 
Visibility 




Fig.l. NESC 5x5 Risk Matrix Scorecard 
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Fig. 2. Current NARMA Consequence Scoring 
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Fig. 3. Proposed NARMA Consequence Scoring 




Table I: Likelihood Level Corresponding Event Examples 


Event 

Likelihood per 

operational 

opportunity 

Level 

Source 

Fatal crashes / passenger airline 
departure 

4.5x1 O' 7 

Level 1 

6 

Fatalities / Public Bus trip 

3.5 x 10" 5 

Level 2 

7 

Fatalities / passenger vehicle trip 

7.5xl0" 5 

Level 2 

8 

Fatalities / motorcycle trip (20 
mile avg trip) 

6.6xl0‘ 4 

Level 2 

9 

Fatalities / motorcycle trip (50 
mile avg trip) 

1.65xl0- 3 

Level 3 

9 

Failures of ELV per launch 

4.5 x 10- 2 

Level 4 

10 

Mortality Rate Open Heart 
Surgery 

2.2 x 10- 2 

Level 4 

11 

Aortic Replacement Surgery 

6.1 x 10' 2 to 

1.02 x 10 1 

Borderline 
Level 4/5 

12, 13 

Mortality Brain Surgery 
(Craniectomies) 

2.2 x 10 1 

Level 5 

14,15 
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